Neural network approximation

Neural networks (NNs) are the method of choice for building learning algorithms. They now being investigated other numerical tasks such as solving high-dimensional partial differential equations. Their popularity stems from their empirical success on several challenging problems (computer chess/Go, autonomous navigation, face recognition). However, most scholars agree that a convincing theoretical explanation this is still lacking. Since these applications revolve around approximating an unknown function data observations, part answer must involve ability NNs to produce accurate approximations. This article surveys known approximation properties outputs with aim uncovering not present in more traditional methods used analysis, approximations using polynomials, wavelets, rational functions and splines. Comparisons made viewpoint rate distortion, i.e. error versus number parameters create approximant. Another major component analysis computational time needed construct approximation, turn intimately connected stability algorithm. So large put forward. The survey, part, concerned popular ReLU activation function. In case piecewise linear rather complicated partitions domain f into cells convex polytopes. When architecture NN fixed allowed vary, set output parametrized nonlinear manifold. It shown manifold has certain space-filling leading increased approximate (better distortion) but at expense stability. space filling creates challenge finding best or good parameter choices when trying approximate.